Object control system and object control method

ABSTRACT

A feeling deduction unit  100  deduces a user&#39;s feeling. An internal state management unit  110  manages an internal state of an object and an internal state of a user on the basis of the deduced user&#39;s feeling. An action management unit  120  determines an action of the object on the basis of the internal state of the object. An output processing unit  140  causes the object to perform the action determined by the action management unit  120.

TECHNICAL FIELD

The present invention relates to technology for controlling an action ofan object that is an actual object or a virtual object.

BACKGROUND ART

In order to construct a good relationship with other people, it is saidthat a “joint viewing experience” in which people see the same thing iseffective instead of a “face-to-face relationship.” It is known that byseeing the same thing at the same place and empathizing with each other,the distance to the other people is shortened and affinity with theother people is increased.

SUMMARY Technical Problems

The inventor has focused on possibility of utilizing a robot as a jointviewing player who experiences sympathy with a user. For example, it isexpected that the user's affinity with the robot is increased andmotivation for playing a game is enhanced by the robot viewing the gameplay next to the user and being pleased or sad together with the user.Further, regarding not only the game but also a movie, a televisionprogram, or the like, it is expected that the user may enjoy contentmere by viewing the content with the robot as compared to the case ofviewing it alone.

The present invention has been made in view of these problems, and anobject of the present invention to provide a technology that allows theuser to obtain a joint viewing experience with the object such as arobot.

Solution to Problems

In order to solve the above problems, an aspect of the present inventionis an object control system for controlling an object, the objectcontrol system including: a feeling deduction unit configured to deducea user's feeling; a user internal state storage unit configured to storean internal state of the user including the user's feeling; an objectinternal state storage unit configured to store an internal state of theobject including an object's feeling; an internal state management unitconfigured to manage the internal state of the object and the internalstate of the user on the basis of the deduced user's feeling; an actionmanagement unit configured to determine an action of the object on thebasis of the internal state of the object; and an output processing unitconfigured to cause the object to perform the action determined by theaction management unit.

Another aspect of the present invention is an object control method. Themethod includes; a step of deducing a user's feeling; a step of managingan internal state of the user including the user's feeling; a step ofmanaging an internal state of an object including an object's feeling; astep of updating the internal state of the object and the internal stateof the user on the basis of the deduced user's feeling; a step ofdetermining an action of the object on the basis of the internal stateof the object; and a step of causing the object to perform thedetermined action.

In addition, it is also effective as an aspect of the present inventionto convert any combination of the above-described components and arepresentation of the present invention into any of a method, anapparatus, a system, a computer program, a recording medium in which thecomputer program is readably recorded, a data structure, etc.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a diagram of an example of appearance at an actualobject.

FIG. 2 illustrates a diagram for explaining an outline of an objectcontrol system.

FIG. 3 illustrates a diagram of an input/output system of a robot.

FIG. 4 illustrates a diagram of a configuration of the object controlsystem that controls an object.

FIG. 5 illustrates a diagram at an example of an appearance form of ahead mounted display (HMD).

DESCRIPTION OF EMBODIMENTS

An object control system of an embodiment provides a mechanism forachieving a joint viewing experience with an actual object or a virtualobject. The actual object may be a human-type pet-type robot, and ispreferable to be able to output at least audio and to move an arm, aleg, a neck, etc., by having ca motor at a joint portion. A user placesa robot nearby and views contents together, while the robot communicateswith the user by outputting a reaction to empathize with the user oroutputting a reaction against the user conversely, on the basis of adeduced user's feeling.

The virtual object, may be a character such as a person or a petincluding a three-dimensional (3D) model, and exists in a virtual spacegenerated by a computer. In the embodiment, in the virtual spaceconstructed when the user wears an HMD, a mechanism is proposed in whichthe content is reproduced in front of the user, and when the user turnssideways, the user may see how the virtual character is viewing thecontent together with the user. Similarly to the robot, the virtualcharacter also communicates with the user by outputting the reaction toempathize with the user or outputting the reaction against the userconversely.

FIG. 1 illustrates a diagram of an example of appearance of the actualobject. This object is a human-type robot 20, and includes a speaker foroutputting audio, a microphone for inputting audio of the outside world,and a drive mechanism that includes a motor for moving each joint or alink for connecting the motors, etc. The robot 20 preferably has aninteractive function to talk with the user, and more preferably has anautonomous movement function.

FIG. 2 illustrates a diagram for explaining an outline of the objectcontrol system 1. FIG. 2 illustrates a situation in which the user sitson a sofa playing the game and the robot 20 sits on the same sofaviewing the user playing the game. Ideally, the robot 20 has such anadvanced autonomous movement function as to be seated on the sofa byitself. If the robot 20 does not have such a function, the user carriesthe robot 20 on the sofa and causes the robot 20 to sit next to theuser. The robot 20 may participate in the game as a virtual player andfor example, may virtually operate the opponent team of a baseball gamebeing played by the user.

An information processing apparatus 10 receives operation informationbeing input to an input apparatus 12 by the user, and performs anapplication such as a game. Note that the information processingapparatus 10 may reproduce content media such as a digital versatiledisc (DVD), or may connect a network to reproduce streaming content fromthe content server. A camera 14 is a stereo camera, and photographs theuser sitting in front of a display apparatus 11 which is a television ata predetermined cycle, and provides a photographed image to theinformation processing apparatus 10.

The object control system 1 controls an action of the robot 20 that isan actual object, by deducing the feeling from the reaction of the user.Basically, the action of the robot 20, which supports the game play ofthe user, is controlled so as to be pleased with the user for expressingsympathy for the user, and to provide the user with the joint viewingexperience. Note that the object control system 1 of the embodimentmanages the internal states of the robot 20 and the user. For example,if the robot 20's popularity rating for the user is low, the objectcontrol system 1 controls the action so as not to be pleased (nosympathy) together even when the user is pleased. Note that, in a casein which the robot 20 virtually participates in the game as an opponentplayer of the user, the action may be controlled so as to be sadconversely when the user is pleased. Basically, the robot 20 expressesfeeling by contents of the speech indicating feeling, but may expressfeeling by moving its body.

FIG. 3 illustrates a diagram of an input/output system of the robot 20.A control unit 30 is a main processor that processes and outputs variousdata such as audio data and sensor data, and an instruction. Amicrophone 22 collects surrounding audio and converts it into an audiosignal, and a camera 24 photographs the surrounding and acquires aphotographed image. A storage unit 26 temporarily stores data andinstructions processed by the control unit 30. A communication unit 32transmits data being output from the control unit 30 to the externalinformation processing apparatus 10 by wireless communication via anantenna. The communication unit 32 also receives the audio data anddrive data from the information processing apparatus 10 by wirelesscommunication via the antenna, and outputs the data to the control unit30.

When receiving the audio data, the control unit 30 provides the audio toa speaker 36 and causes the speaker 36 to output the audio. Whenreceiving the drive data, the control unit 30 causes a motor of a drivemechanism 34 to be rotated. The drive mechanism 34 includes the motorincorporated in the joint portion that is a movable part of the robot 20and a link mechanism for connecting the motors. The arms, the leg, theneck, etc. of the robot 20 are moved by driving the motor.

FIG. 4 illustrates a diagram of a configuration of the object controlsystem that controls the object. The object control system 1 includes afeeling deduction unit 100, an internal state management unit 110, anaction management unit 120, an internal state storage unit 130, and anoutput processing unit 140. The internal state storage unit 130 has anobject internal state storage unit 132 and a user internal state storageunit 134. The feeling deduction unit 100 includes a first feelingdeduction unit 102, a second feeling deduction unit 104, and a thirdfeeling deduction unit 106, and performs a process of deducing feelingswith these three systems.

The object control system 1 also includes the camera 14 forphotographing the user, a biological sensor 15 for detecting biologicalinformation of the user, a motion sensor 16 for detecting a motion ofthe user, a microphone 17 for acquiring audio signals around the user,and an event detection unit 40 for detecting occurrence of an event.

In FIG. 4 , each element, being described as a functional block thatperforms various processing, may include, a circuit block, a memory,other large-scale integrated circuits (LSIs) in terms of hardware, andmay be achieved by a program loaded into the memory, etc., in terms ofsoftware. Therefore, it is understood by the person skilled in the artthat these functional blocks may be achieved in various forms by onlyhardware, only software, or a combination thereof, and is not limited toany of them.

In FIG. 4 , each configuration of the feeling deduction unit 100, theinternal stare management unit 110, the action management unit 120, theinternal state storage unit 130, and the output processing unit 140 isprovided in the information processing apparatus 10, and the Informationprocessing apparatus 10 may control the action of the robot 20. Further,each configuration of the feeling deduction unit 100, the internal statemanagement unit 110, the action management unit 120, the internal statestorage unit 130, and the output processing unit 140 is provided in therobot 20, and the robot 20 may control its own action autonomously. Inthis case, the robot 20 includes an autonomous robot and may act alone.In addition, a part of the feeling deduction unit 100, the internalstate management unit 110, the action management unit 120, the internalstate storage unit 130, and the output processing unit 140 may beprovided in the information processing apparatus 10, and the rest may beprovided in the robot 20. In this case, the information processingapparatus 10 and the robot 20 cooperate with each other and operate tocontrol the action of the robot 20.

Thus, the object control system 1 illustrated in FIG. 4 may be achievedin various manners. In a case in which each of the above-describedconfigurations is provided in the information processing apparatus 10,it is sufficient that the audio signal acquired by the microphone 22 ofthe robot 20 and the photographed image that is photographed by thecamera 24 may be transmitted to the information processing apparatus 10for use in processing in the information processing apparatus 10.Further, in a case in which each of the above-described configurationsis provided in the robot 20, it is sufficient that the audio signalacquired by the microphone 17 connected to the information processingapparatus 10 and the photographed image that is photographed by thecamera 14 may be transmitted to the robot 20 for use in processing inthe robot 20. In addition, it is sufficient that the audio signal or thephotographed image may be acquired by a microphone or a camera otherthan illustrated and may be provided to the information processingapparatus 10 or the robot 20. In the following, for convenience ofexplanation, as a representative, the camera 14 photographs the user,and the microphone 17 collects the user's utterance and converts it intothe audio signal.

The feeling deduction unit 100 deduces the user's feeling on the basisof the outputs of various sensors and data from an external server. Thefeeling deduction unit 100 deduces the user's feeling by deriving anevaluation value for each of feeling indexes such as joy, anger, love,and surprise. In the embodiment, the user's feeling is expressed by asimple model, and the feeling deduction unit 100 deduces the user'sfeeling from three evaluation values: good feeling (positive feeling),bad feeling (negative feeling), and neither good nor bad feeling(neutral feeling). Note that in a practical use, each of “positive” and“negative” may be subdivided into a plurality of stages, allowing fineevaluation values to be derivable.

The first feeling deduction unit 102 deduces the user's feeling byperforming nonverbal action understanding based on paralinguisticinformation or nonverbal information on the basis of outputs of varioussensors, i.e., outputs of the camera 14 that is an image sensor, thebiological sensor 15, the motion sensor 16, and the microphone 17. Ascompared with the second feeling deduction unit 104, the first feelingdeduction unit 102 is characterized in that the user's feeling isdeduced not being on the basis of the contents of the speech.

The first feeling deduction unit 102 extracts changes in facialexpression, a gaze direction, a hand gesture, etc., of the user from thephotographed image of the camera 14 and deduces the user's feeling. Thebiological sensor 15 is attached to the user and detects biologicalinformation such as a user's heart rate and user's sweating state. Thefirst feeling deduction unit 102 derives the user's mental state fromthe biological information and deduces the user's feeling. The motionsensor 16 is attached to the user and detects the user's motion, and thefirst feeling deduction unit 102 deduces the user's feeling from theuser's motion. Note that a role of the motion sensor 16 may also besubstituted by analyzing the photographed image of the camera 14.Further, the first feeling deduction unit 102 deduces the user's feelingfrom the audio signal of the microphone 17 using a feature amount of theparalinguistic information. The paralinguistic information includesinformation such as speech speed, volume, voice inflection, intonation,wording, etc. The first feeling deduction unit 102 collects the user'sfeeling deduced from each of the outputs of the various sensors as oneevaluation value for each evaluation index, and provides the evaluationvalue to the internal state management unit 110.

The second feeling deduction unit 104 deduces the user's feeling on thebasis of the contents of the user's speech. More specifically, thesecond feeling deduction unit 104 performs natural languageunderstanding for audio analysis of the contents of the user's speechfrom the output of the microphone 17, and then, deduces the user'sfeeling. Known algorithms may be used for natural language understandingtechniques. If the user speaks that “I did it. I've got a home run”while playing the baseball game, the second feeling deduction unit 104may deduce the feeling of the “joy” index of the user as “positive,”whereas if the user speaks that “I have given up a home run,” the secondfeeling deduction unit 104 may deduce the feeling of the “joy” index ofthe user as “negative.” The second feeling deduction unit 104 providesthe evaluation value of the deduced user's feeling to the internal statemanagement unit 110.

The event detection unit 40 detects occurrence of the event in theobject control system 1 and notifies the third feeling deduction unit106 of contents of the occurred event. Regarding a game event, byperforming the game program on an emulator, the emulator may detect anevent such as a home run, so that the event detection unit 40 may benotified of the event from the emulator. Note that the event detectionunit 40 may detect a home run event by referring to an access toperformance data being reproduced at the time of a home run, or by theperformance actually displayed on the display apparatus 11. The thirdfeeling deduction unit 106 deduces the user's feeling from the notifiedevent contents, and provides the evaluation value to the internal statemanagement unit 110.

Note that the event detection unit 40 may be provided with an eventoccurrence timing from the external server that stores big data. Forexample, in a case in which the information processing apparatus 10reproduces movie content, the event detection unit 40 acquires, inadvance from the external server, a correspondence relationship tablebetween time information of a scene of interest of the movie content anda deduced user's feeling in the scene of interest. The informationprocessing apparatus 10 provides the correspondence relationship tableto the third feeling deduction unit 106. When the time of the scene ofinterest arrives, the event detection unit 40 notifies the third feelingdeduction unit 106 of the time information. The third feeling deductionunit 106 refers to the correspondence relationship table to acquire theuser's feeling being associated with the time information, and deducesthe user's feeling in the scene of interest. As described above, thethird feeling deduction unit 106 may deduce the user's feeling on thebasis of the information from the external server.

In the internal state storage unit 130, the object internal statestorage unit 132 stores an internal state of the robot 20 that is anobject, and the user internal state storage unit 134 stores an internalstate of the user. The internal state of the robot 20 is defined atleast by the feeling of the robot 20 and the robot 20's popularityrating for the user, and the internal state of the user is defined bythe user's feeling and the user's popularity rating for the robot 20.The popularity rating is generated on the basis of evaluation values ofa plurality of the latest feelings up to the present, and is evaluatedby a long-term relationship between the user and the robot 20. Theinternal state of the robot 20 is derived from the speech and action ofthe user with respect to the robot 20.

The internal state management unit 110 manages the internal state of therobot 20 and the internal state of the user on the basis of the user'sfeeling deduced by the feeling deduction unit 100. Hereinafter, theinternal state of the user will be described first, and then, theinternal state of the robot 20 will be described. The “feeling” of theuser is set by the feeling deduced by the feeling deduction unit 100.The internal state management unit 110 updates the evaluation value inthe user internal state storage unit 134 on the basis of the feelingdeduced by the feeling deduction unit 100.

When the evaluation value of “positive” is provided from the feelingdeduction unit 100 for the feeling index of the user, the internal statemanagement unit 110 updates the evaluation value of the feeling index to“positive,” and when the evaluation value of “negative” is provided, theinternal state management unit 110 updates the evaluation value of thefeeling index to “negative.” Here, “update” means a process ofoverwriting the original evaluation value in the internal state storageunit 130. Even in a case in which the original evaluation value is“positive” and the evaluation value to be overwritten is “positive,”i.e., there is no change in the evaluation value, such case is called to“update” the evaluation value.

Note that the feeling deduction unit 100 deduces the user's feeling withthe three systems including the first feeling deduction unit 102, thesecond feeling deduction unit 104, and the third feeling deduction unit106, so that it may be possible that the evaluation values of the user'sfeeling deduced in each system contradict each other. Hereinafter,evaluation value update processing regarding the “joy” index when theuser gets a home run in the baseball game will be described.

The third feeling deduction unit 106 holds an evaluation value table fordeducing feeling as “positive” in a case in which the user gets a homerun and as “negative” in a case in which the user gives up a home run.Upon receiving the event notification that the user has got a home runfrom the event detection unit 40, the third feeling deduction unit 106deduces the evaluation value of the “joy” index as “positive” withreference to the evaluation value table.

Usually, when the user gets a home run, the user expresses feeling ofthe joy such as moving the body and chattering or saying, “I did it.I've got a home run.” At this time, the first feeling deduction unit 102and the second feeling deduction unit 104 both deduce the evaluationvalue of the “joy” index as “positive.” Therefore, the evaluation valuesof the ‘joy’ index provided to the internal state management unit 110from the feeling deduction unit 100 via the three systems are all“positive,” and the internal state management unit 110 updates theevaluation value of the “joy” index in the user internal state storageunit 134 to “positive.”

However, in a case in which the user is aiming for anything other than ahome run, the user may not always express the speech and action of joy.For example, when the user is aiming for a triple hit in order toestablish a cycle hit (one batter will get an one-base hit, a two-basehit, a three-base hit, and a home run in one game), the user may not bepleased even if the user get a home run. In the game, besides winningand losing, there are also provided rules for commending a special playsuch as a cycle hit, in many cases. Accordingly, a user who is aimingfor awards may be happier with a three-base hit than a home run. Thus,when the user who got a home run speaks “Oops. I've got a home run”dispiritedly, the first feeling deduction unit 102 and the secondfeeling deduction unit 104 both deduce the evaluation value of the “joy”index as “negative.”

Therefore, the internal state management unit 110 is provided with the“negative” evaluation value from the first feeling deduction unit 102and the second feeling deduction unit 104, and the “positive” evaluationvalue from the third feeling deduction unit 106. The internal statemanagement unit 110 may adopt an evaluation value that matches with halfor more of the evaluation values being provided from the three systems.That is, the internal state management unit 110 may adopt an evaluationvalue that matches with half or more of the evaluation values beingindependently provided from a plurality of systems, according to amajority rule. Therefore, in this case, the evaluation value of the“joy” index may be updated to “negative.”

Note that the internal state management unit 110 may set priorities tothe evaluation values provided from the three systems, and may determinethe evaluation value to be adopted according to the priorities. Theranking of each evaluation value will be described below. First, theevaluation value deduced by the third feeling deduction unit 106 is aunique evaluation value defined for the event, and does not reflect theactual state of the user. Therefore, the evaluation value deduced by thethird feeling deduction unit 106 may be set to the lowest priority to beadopted.

Next, when the first feeling deduction unit 102 and the second feelingdeduction unit 104 are compared, the second feeling deduction unit 104deduces feeling on the basis of only the contents of the user's speech,and there is an aspect that an amount of information for the deductionis small. In addition, even if the contents of the speech themselves arepositive contents, the user may speak self-abusively, in some cases. Forexample, there are some cases where although the user was aiming for athree-base hit, the user speaks “I've got a home run” dropping theuser's shoulder dispiritedly. In this case, the second feeling deductionunit 104 deduces the “positive” evaluation value, but the first feelingdeduction unit 102 deduces the “negative” evaluation value from theuser's attitude or the paralinguistic information. As described above,since the user may speak a word contrary to emotion, the internal statemanagement unit 110 may set reliability of the evaluation value by thefirst feeling deduction unit 102 higher than reliability of theevaluation value by the second feeling deduction unit 104. Therefore,the internal state management unit 110 may set the priority in the orderof the evaluation values according to the first feeling deduction unit102, the second feeling deduction unit 104, and the third feelingdeduction unit 106.

The internal state management unit 110 may update the internal state ofthe user by setting the priority if the evaluation value may be obtainedby at least one system. In a case in which the user's feeling deduced bythe first feeling deduction unit 102 does not match with the user'sfeeling deduced by the second feeling deduction unit 104, the internalstate management unit 110 updates the internal state of the user on thebasis of the user's feeling deduced by the first feeling deduction unit102 according to the priority. Also, for example, when the user gets ahome run, in a case in which the first feeling deduction unit 102 andthe second feeling deduction unit 104 may not deduce the positive ornegative user's feeling (in a case in which the user neither moves norspeaks at all), it is sufficient that the internal state management unit110 adopts the evaluation value deduced by the third feeling deductionunit 106.

The user's “popularity rating” for the robot 20 is derived from aplurality of the evaluation values of the feeling deduced from thespeech and action performed by the user on the robot 20 up until thepresent time. A name of the robot 20 is assumed to be “Hikoemon” in thefollowing description. When the robot 20 speaks to the user, if the userresponds as “Hikoemon, that's exactly what you say,” the feelingdeduction unit 100 deduces that the evaluation value of the user's“love” index is “positive,” whereas if the user responds as “Hikoemon,you're noisy, be quiet,” the feeling deduction unit 100 deduces that theevaluation value of the user's “love” index is “negative.” Also, whenthe robot 20 speaks to the user as “Please charge me soon,” if the usercharges the robot 20 immediately, the feeling deduction unit 100 deducesthat the evaluation value of the user's “love” index to “positive,”whereas if the user does not charge immediately, the feeling deductionunit 100 deduces that the evaluation value of the user's “love” index to“negative.” Furthermore, if the user strokes the head of the robot 20,the feeling deduction unit 100 deduces that the evaluation value of theuser's “love” index is “positive,” whereas if the user kicks the robot20, the feeling deduction unit 100 deduces that the evaluation value ofthe user's “love” index to “negative.”

When receiving the evaluation value of the user's “love” index from thefeeling deduction unit 100, the internal state management unit 110reflects the evaluation value on the user's popularity rating. Theinternal state management unit 110 stocks the plurality of theevaluation values of the “love” index up until the present time, anddeduces, for example, the user's popularity rating for the robot 20 onthe basis of the plurality of the latest evaluation values (e.g., 21evaluation values). Here, among the plurality of the evaluation valuesexcluding the neutral evaluation value, the evaluation value having thelarger number of the positive evaluation value and the negativeevaluation value may be set as the evaluation value of the popularityrating. That is, if there are 11 or more positive evaluation valuesamong the latest 21 evaluation values, the evaluation value of thepopularity rating may be set as “positive,” whereas if there are 11 ormore negative evaluation values, the evaluation value of the popularityrating may be set as “negative.”

The above is the description of the internal state of the user, and thenthe internal state of the robot 20 will be described.

The robot 20's “popularity rating” for the user is derived from theplurality of the evaluation values of the feeling deduced from thespeech and action performed by the user on the robot 20 up until thepresent time. The internal state management unit 110 receives theevaluation value of the “love” index from the feeling deduction unit100, and reflects the evaluation value on the robot 20's popularityrating for the user. The robot 20's popularity rating for the user tendsto be linked to the user's popularity rating for the robot 20 describedabove. The internal state management unit 110 stocks the plurality ofthe evaluation values of the “love” index up to the present time, anddetermines the robot 20's popularity rating for the user on the basisof, for example, the plurality of the latest evaluation values (e.g., 11evaluation values). Here, the reason for reducing the number of theevaluation values to be referred to when deducing the user's popularityrating is to make it easy to change the robot 20's popularity rating forthe user. The robot 20's popularity rating for the user is an importantparameter in determining the robot 20's feeling. Note that thepopularity rating for the user may be evaluated not only by “positive”and “negative” but also by each of “positive” and “negative” in theplurality of stages.

In the embodiment, if the robot 20's popularity rating for the user ispositive, the robot 20 empathizes with the user and performs action toachieve the joint viewing experience, while if the popularity rating forthe user is negative, the robot 20 does not empathize with the user, butrather performs the action to react against the user. The actionmanagement unit 120 determines the action including the audio output ofthe robot 20 on the basis of the internal state of the robot 20 and theinternal state of the user.

The action of the robot 20 when the user gets a home run in the baseballgame will be described. The feeling deduction unit 100 deduces that theevaluation value of the “joy” index is “positive,” and the internalstate management unit 110 updates the evaluation value of the “joy”index of the user to “positive.” Further, the internal state managementunit 110 determines the evaluation value of the “joy” index of the robot20 with reference to the evaluation value of the “joy” index of the userand the evaluation value of the robot 20's popularity rating for theuser.

<In a Case in Which the Popularity Rating for the User is Positive>

In the case that the robot 20's popularity rating for the user is set tobe positive, the internal state management unit 110 manages the robot20's feeling so as to empathize with the user's feeling. Therefore, ifthe evaluation value of the “joy” index of the user is updated to bepositive, the internal state management unit 110 also updates theevaluation value of the “joy” index of the robot 20 to be positive. Uponupdating the internal state of the robot 20, the internal statemanagement unit 110 provides the action management unit 120 with triggerinformation indicating that it is time to determine the action of therobot 20. Upon receiving the trigger information, the action managementunit 120 determines the action of the robot 20 including the audiooutput on the basis of the internal state of the robot 20 updated in theobject internal state storage unit 132, specifically, on the basis ofthe evaluation value of the updated feeling. For example, the actionmanagement unit 120 generates the contents of the speech “o×(player'sname) is awesome. He has got a home run,” and determines the action asthe movement of the robot 20 with arms up in the air twice. Thesecontents of the speech and the movements are notified to the outputprocessing unit 140, and the output processing unit 140 causes the robot20 to perform the action determined by the action management unit 120.

The output processing unit 140 generates the audio data for controllingthe output timing of the robot 20 such as a speech speed and each timingof the speech and the movement, etc., and the drive data for driving therobot 20, to output the data to the robot 20. When the communicationunit 32 in the robot 20 receives the audio data and the drive data, thecontrol unit 30 provides the audio data to the speaker 36 so as to causethe speaker 36 to output the audio, and cause the motor of the drivemechanism 34 to be driven on the basis of the drive data. Accordingly,the robot 20 performs the speech and the movement that empathizes withthe user who has got a home run. The user sees the robot 20 beingpleased together, and the affinity with the robot 20 as well as themotivation for the game may be increased.

Conversely, in a case in which the user gives up a home run, theevaluation value of the “joy” index of the user is updated to negative,so that the internal state management unit 110 updates the evaluationvalue of the “joy” index of the robot 20 to negative. Upon updating theinternal state of the robot 20, the internal state management unit 110provides the action management unit 120 with the trigger information.Thereby, the action management unit 120 determines the action includingthe audio output of the robot 20 on the basis of the evaluation value ofthe “joy” index of the robot 20 updated in the object internal statestorage unit 132. For example, the action management unit 120 generatesthe contents of the speech “Oh no. You've given up a home run,” anddetermines, as the movement of the robot 20, the action that causesweakness in a dejected manner. These contents of the speech andmovements are notified to the output processing unit 140, and the outputprocessing unit 140 causes the robot 20 to perform the action determinedby the action management unit 120.

<In a Case in Which the Popularity Rating for the User is Negative>

In contrast, in a case in which the robot 20's popularity rating for theuser is set to be negative, the internal state management unit 110manages the robot 20's feeling so as not to empathize with the user'sfeeling. The internal state management unit 110 rather manages the robot20's feeling so as to react against the user, thereby allowing the userto realize that the user should treat the robot 20 more carefully.

Therefore, even if the evaluation value of the “joy” index of the userwho has got a home run is updated to positive, the internal statemanagement unit 110 updates the evaluation value of the “joy” index ofthe robot 20 to negative. If the robot 20's popularity rating for theuser is negative, the internal state management unit 110 updates theevaluation value of the robot 20's feeling to a value opposite to theevaluation value of the user's feeling. Therefore, if the feelingevaluation value of the user is positive, the feeling evaluation valueof the robot 20 may updated negative. If the feeling evaluation value ofthe user is negative, the feeling evaluation value of the robot 20 maybe updated positive.

Upon updating the internal state of the robot 20, the internal statemanagement unit 110 provides the action management unit 120 with triggerinformation indicating that it is time to determine the action of therobot 20. Accordingly, the action management unit 120 determines theaction including the audio output of the robot 20 on the basis of theinternal state of the robot 20 updated in the object internal statestorage unit 132, specifically, the negative evaluation value of the“joy” index. For example, the action management unit 120 may notgenerate the contents of the speech (i.e., ignore the user), and maydetermine the action as the movement of the robot 20 to turn in theopposite direction to the user (turn to the other side). These contentsof the action are notified to the output processing unit 140, and theoutput processing unit 140 causes the robot 20 to perform the actiondetermined by the action management unit 120.

Seeing this action of the robot 20, the user notices that the robot 20is not pleased with the user this time, although the robot 20 waspleased together before. The user looks back at the user's own attitudeand recognises that the robot 20 is not pleased together with the userbecause of the user's cold contact with the robot 20. By causing therobot 20 to react against the user by dare, an opportunity for the userto keep in mind to gently treat the robot 20 from now on is given to theuser.

Thus, in the object control system 1, the relationship between the userand the robot 20 is managed in the same manner as the human relationshipin an actual human society. In the communication between people in realworld, a good relationship is generated by coming in contact with eachother with consideration for each other. If one is not considerate, theother will not be able to be considerate. In the object control system1, this considerateness is expressed by the feeling of the “love” index,and the human relationship through the considerateness is expressed bythe “popularity rating” index. Therefore, the evaluation value of theuser's popularity rating in the object control system 1 and theevaluation value of the robot 20's popularity rating also tend to belinked. In a case in which the robot 20's popularity rating for the useris lowered, the evaluation value for the popularity rating for the useris improved from negative to positive by the user coming in contact withthe robot 20 with more consideration, and then, the user may get againthe joint viewing experience together with the robot 20.

The robot 20 may be a kind of a friend by increasing the user's affinitywith the robot 20. If the user is living an irregular life, the robot 20may propose improvement of the life rhythm, for example, saying “Let'sgo to bed soon” and such a case that the user listens to and accepts therobot 20 as advice from a friend may be also considered in the future.In order to achieve such case described above, constructing a mechanismto increase the user's affinity with the robot 20 through the jointviewing experience with the robot 20 also leads to expanding the futurepossibilities of the robot 20.

Note that, in the above-described embodiment, although the mechanism isproposed in which the user obtains the joint viewing experience with therobot 20 being an actual object, a mechanism for obtaining the jointviewing experience between the virtual objects may be similarlyconstructed.

In this case, the virtual object may be a character such as a person ora pet including the 3D model, and exists in the virtual space generatedby the computer. In the embodiment, in the virtual space constructedwhen the user wears the HMD, the mechanism is proposed in which thecontents are reproduced in front of the user, and when the user turnssideways, the user may see how the virtual character is viewing thecontent together with the user. Similarly to the robot 20, the virtualcharacter also communicates with the user by outputting the reaction toempathize with the user or outputting the reaction against the userconversely.

FIG. 5 illustrates a diagram of an example of the appearance form of theHMD 200. The HMD 200 includes an output mechanism unit 202 and amounting mechanism unit 204. The mounting mechanism unit 204 includes amounting band 206 for fixing the HMD 200 to the head by circling thehead with being worn by the user.

The output mechanism unit 202 includes a housing 208 shaped to cover theleft and right eyes in a state where the user wears the HMD 200, andinternally includes a display panel which is located opposed to the eyesat the time of wearing. The display panel may be a liquid crystal panelor an organic electroluminescence (EL) panel. The housing 208 furtherincludes a pair of left and right optical lenses located between thedisplay panel and the user's eyes and enlarging a viewing angle of theuser. The HMD 200 may further include a speaker or an earphone at aposition corresponding to the user's ear and may be configured to beconnected with an external headphone. In addition, a camera thatphotographs the face of the user is provided inside the housing 208 andis used to detect a facial expression of the user.

On an outer surface of the housing 208, light emission markers 210 a,210 b, 210 c, and 210 d are provided, which are tracking light emittingdiodes (LEDs). The light emission marker 210 is photographed by thecamera 14, and the information processing apparatus 10 analyzes aposition of each marker. Further, a posture sensor (acceleration sensorand gyro sensor) is mounted on the HMD 200. The HMD 200 is connected tothe information processing apparatus 10 by a known wirelesscommunication protocol, and transmits the sensor data detected by theposture sensor to the information processing apparatus 10. Theinformation processing apparatus 10 identifies a direction in which theHMD 200 faces on the basis of the photographed position of the lightemission marker 210 and the sensor data of the posture sensor.

Here, the information processing apparatus 10 transmits content datasuch as a movie to the HMD 200. The information processing apparatus 10sets a reference direction in which the HMD 200 faces, and installs avirtual display in the reference direction, and then, displaysreproducing data of content on the virtual display. Further, theinformation processing apparatus 10 arranges the virtual object facingthe virtual display at a position rotated by 90 degrees with respect tothe reference direction, for example, around the viewpoint position.Accordingly, when the user rotates the gaze direction sideways by 90degrees, the user may see the virtual object viewing content togethernext to the user.

In this way, the user wearing the HMD 200 may obtain the joint viewingexperience with the virtual object. The virtual object actiondetermination process is as described for the robot 20. In addition,since the degree of freedom of action of the virtual object is higherthan that for the robot 20, the various positive or negative feeling maybe caused to be expressed.

As described above, according to the embodiment, the user may increasethe affinity with the actual object or the virtual object through thejoint viewing experience, thereby the user's life may be furtherenriched.

The present invention has been described above on the basis of theembodiments. The embodiment is an exemplification, and it is understoodby the person skilled in the art that various modifications may be madeto the combinations of the respective constituent elements and therespective processing processes and such modifications are also withinthe scope of the present invention.

In the embodiment, the action management unit 120 generates the contentsof the speech, “o×(player's name) is awesome. He has got a home run” asan empathic expression when the user gets a home run. In the modifiedexample, in order to emphasize the joint viewing experience, the actionmanagement unit 120 may actively cause the object to output the audio inwhich a subject is replaced by a demonstrative pronoun. Specifically,the expression of “o×(player's name)” may be replaced with “He,” whichis the demonstrative pronoun, to generate the contents of the speechsuch as “He is awesome. He has got a home run.” By using thedemonstrative pronoun, the user may further increase a sense of viewingtogether with the robot 20.

REFERENCE SIGNS LIST

1 . . . Object control system, 10 . . . Information processingapparatus, 20 . . . Robot, 40 . . . Event detection unit, 100 . . .Feeling deduction unit, 102 . . . First feeling deduction unit, 104 . .. Second feeling deduction unit, 106 . . . Third feeling deduction unit,110 . . . Internal state management unit, 120 . . . Action managementunit, 130 . . . Internal state storage unit, 132 . . . Object internalstate storage unit, 134 . . . User internal state storage unit, 140 . .. Output processing unit.

INDUSTRIAL APPLICABILITY

The present invention may be used in the technical field in which anaction of an object is controlled.

The invention claimed is:
 1. An object control system controlling anobject, the object control system comprising: a feeling deduction unitconfigured to deduce a user's feeling; a user internal state storageunit configured to store an internal state of the user including theuser's feeling; an object internal state storage unit configured tostore an internal state of the object including an object's feeling; aninternal state management unit configured to manage the internal stateof the object and the internal state of the user on a basis of thededuced user's feeling; an action management unit configured todetermine an action of the object on a basis of the internal state ofthe object; and an output processing unit configured to cause the objectto perform the action determined by the action management unit, whereinthe internal state of the object includes an object's popularity ratingfor the user being derived from a plurality of evaluation values of afeeling that is deduced up until the present time, and wherein theaction of the object is selected to be in alignment with the user'sfeeling if the popularity rating is greater than a predeterminedthreshold, and wherein the action of the object is selected to be inopposition to the user's feeling if the popularity rating is less thanthe predetermined threshold.
 2. The object control system according toclaim 1, wherein the feeling deduction unit includes: a first feelingdeduction unit configured to deduce the user's feeling on a basis ofsensor output; and a second feeling deduction unit configured to deducethe user's feeling on a basis of contents of a user's speech, and theinternal state management unit updates the internal state of the objectand/or the internal state of the user on a basis of output of the firstfeeling deduction unit and/or the second feeling deduction unit.
 3. Theobject control system according to claim 2, wherein in a case in whichthe user's feeling deduced by the first feeling deduction unit does notmatch with the user's feeling deduced by the second feeling deductionunit, the internal state management unit updates the internal state ofthe user on a basis of the user's feeling deduced by the first feelingdeduction unit.
 4. The object control system according to claim 1,wherein the object includes an actual object or a virtual object.
 5. Theobject control system according to claim 1, wherein the actionmanagement unit causes the object to output audio in which a subject isreplaced by a demonstrative pronoun.
 6. An object control methodcomprising: deducing a user's feeling; storing an internal state of theuser including the user's feeling; storing an internal state of anobject including an object's feeling; managing the internal state of theobject and the internal state of the user on a basis of the deduceduser's feeling; determining an object's popularity rating for the userbeing derived from a plurality of evaluation values of a feeling that isdeduced up until the present time; determining an action of the objecton a basis of the internal state of the object; and causing the objectto perform the determined action, wherein the action of the object isselected to be in alignment with the user's feeling if the popularityrating is greater than a predetermined threshold, and wherein the actionof the object is selected to be in opposition to the user's feeling ifthe popularity rating is less than the predetermined threshold.
 7. Anon-transitory computer readable medium having stored thereon a programfor a computer, comprising: by a feeling deduction unit, deducing auser's feeling; by a user internal state storage unit, storing aninternal state of the user including the user's feeling; by an objectinternal state storage unit, storing an internal state of an objectincluding an object's feeling; by an internal state management unit,managing the internal state of the object and the internal state of theuser on a basis of the deduced user's feeling; by an action managementunit, determining an action of the object on a basis of the internalstate of the object; and by an output processing unit, causing theobject to perform the determined action, wherein the internal state ofthe object includes an object's popularity rating for the user beingderived from a plurality of evaluation values of a feeling that isdeduced up until the present time, and wherein the action of the objectis selected to be in alignment with the user's feeling if the popularityrating is greater than a predetermined threshold, and wherein the actionof the object is selected to be in opposition to the user's feeling ifthe popularity rating is less than the predetermined threshold.